Image processing device, image processing method, and program

ABSTRACT

An image processing device according to the present disclosure includes: a movement detection unit that detects movement of a movable writing medium based on moving image data obtained by imaging the writing medium and a writer writing on the writing medium; and a writing portion detection unit that detects a writing portion on the writing medium, wherein the writing portion detection unit executes processing corresponding to the movement of the writing medium in a case where the movement of the writing medium is detected.

FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a program.

BACKGROUND

A technique for extracting a writing portion on a writing medium frommoving image data obtained by imaging a state where a writer is writingon the writing medium has been proposed (for example, see PatentLiterature 1). Such image processing requires further accuracy.

CITATION LIST Patent Literature

Patent Literature 1: WO 2018/163977 A

SUMMARY Technical Problem

The present disclosure provides an image processing device, an imageprocessing method, and a program capable of performing image processingwith high accuracy even in a case where writing is performed on amovable writing medium.

Solution to Problem

An image processing device according to the present disclosure includes:a movement detection unit that detects movement of a movable writingmedium based on moving image data obtained by imaging the writing mediumand a writer writing on the writing medium; and a writing portiondetection unit that detects a writing portion on the writing medium,wherein the writing portion detection unit executes processingcorresponding to the movement of the writing medium in a case where themovement of the writing medium is detected.

Advantageous Effects of Invention

According to the image processing device, the image processing method,and the program according to the present disclosure, image processingcan be performed with high accuracy even in the case where writing isperformed on the movable writing medium.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of animage processing system according to a first embodiment of the presentdisclosure.

FIG. 2 is a block diagram illustrating an example of a functionalconfiguration of an image processing device according to the firstembodiment of the present disclosure.

FIG. 3 is a block diagram illustrating an example of a hardwareconfiguration of the image processing device according to the firstembodiment of the present disclosure.

FIG. 4 is a diagram illustrating a movement detection result of a boardsurface by the image processing device according to the first embodimentof the present disclosure, together with moving image data.

FIG. 5 is a diagram explaining a method of distinguishing between awriting portion and a non-writing portion by the image processing deviceaccording to the first embodiment of the present disclosure.

FIG. 6 is a diagram illustrating a detection result of the writingportion by the image processing device according to the first embodimentof the present disclosure, together with moving image data.

FIG. 7 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device according to the firstembodiment of the present disclosure.

FIG. 8 is a diagram illustrating a movement detection result of a boardsurface by an image processing device according to a first modificationof the first embodiment of the present disclosure, together with movingimage data.

FIG. 9 is a diagram illustrating a movement detection result of a boardsurface by an image processing device according to a second modificationof the first embodiment of the present disclosure, together with movingimage data.

FIG. 10 is a block diagram illustrating an example of a functionalconfiguration of an image processing device according to a secondembodiment of the present disclosure.

FIG. 11 is a diagram illustrating a generation result of writing dataand composite moving image data by the image processing device accordingto the second embodiment of the present disclosure, together with movingimage data.

FIG. 12 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device according to the secondembodiment of the present disclosure.

FIG. 13 is a block diagram illustrating an example of a functionalconfiguration of an image processing device according to a thirdembodiment of the present disclosure.

FIG. 14 is a diagram illustrating a generation result of a writing logby the image processing device according to the third embodiment of thepresent disclosure, together with moving image data.

FIG. 15 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device according to the thirdembodiment of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the drawings. Note that, in the followingembodiments, the same parts are denoted by the same reference numeralsso that redundant description can be omitted.

First Embodiment

A first embodiment will be described with reference to the drawings.

Example of Configuration of Information Processing System

FIG. 1 is a diagram illustrating an example of a configuration of animage processing system 1 according to the first embodiment of thepresent disclosure. As illustrated in FIG. 1 , the image processingsystem 1 of the first embodiment includes an image processing device 10,an imaging device 20, an input/output device 30, and a recording device40.

The imaging device 20 is a video camera or the like that records alecture, training, a seminar, a conference, or the like in which awriting medium such as a blackboard, a whiteboard, or an electronicblackboard is used. In the example of FIG. 1 , the imaging device 20images a state in which a writer 300 is giving an explanation whilewriting on a whiteboard 200.

Here, the writer 300 is a lecturer, a teacher, a presenter, or the likewho gives a lecture, training, a seminar, a conference, or the like.

The whiteboard 200 as a writing medium group includes a plurality ofboard surfaces 210 a, 210 b, 210 c, and 210 d as writing media. Inaddition, the whiteboard 200 itself is fixed to, for example, a wallsurface of a building or the like, but is configured as a slidablewhiteboard in which vertically arranged board surfaces, that is, a boardsurface 210 a and a board surface 210 b, or a board surface 210 c and aboard surface 210 d, are replaceable with each other due to raising andlowering. The writer 300 can write or erase the written content on theindividual board surfaces 210 a to 210 d while raising or lowering theboard surfaces 210 a and 210 b or the board surfaces 210 c and 210 dvertically arranged.

However, the whiteboard 200 may be of a type in which horizontallyarranged board surfaces, that is, the board surface 210 a and the boardsurface 210 c, or the board surface 210 b and the board surface 210 d,are replaced with each other due to lateral sliding. In addition, thenumber of board surfaces 210 a to 210 d that move at one time may beone, or three or more.

The board surfaces 210 a to 210 d may have black, green, or other colorsin addition to white. Characters, graphs, figures, or the like may bewritten on the board surfaces 210 a to 210 d. The writing color may alsobe any color as long as it can be identified on the board surfaces 210 ato 210 d, and a plurality of colors may be used.

Hereinafter, when no distinction is made between the individual boardsurfaces 210 a, 210 b, 210 c, and 210 d, they may be simply referred toas a board surface 210.

The imaging device 20 is set at a position and an angle of view at whichthe writer 300 and the whiteboard 200 can be simultaneously imaged. Theimaging device 20 generates moving image data obtained by imaging thewriter 300 and the whiteboard 200, and outputs the generated movingimage data to the image processing device 10.

The image processing device 10 performs processing of extracting awriting portion from the moving image data generated by the imagingdevice 20. At this time, the image processing device 10 performsprocessing according to raising and lowering movement of the boardsurface 210.

The input/output device 30 is configured such that a user can input aninstruction or the like to the image processing device 10. Further, theinput/output device 30 acquires an extraction result of the writingportion from the image processing device 10 and presents the extractionresult to the user or the like, for example.

The recording device 40 records the moving image data processed by theimage processing device 10, the extraction result of the writingportion, and the like.

Configuration Example of Image Processing Device

FIG. 2 is a block diagram illustrating an example of a functionalconfiguration of the image processing device 10 according to the firstembodiment of the present disclosure.

As illustrated in FIG. 2 , the image processing device 10 is connectedto one or more imaging devices 20, input/output devices 30, andrecording devices 40. The image processing device 10 may be connecteddirectly to the imaging device 20, the input/output device 30, and therecording device 40, or may be connected via a network.

The image processing device 10 receives the moving image data outputfrom one or more imaging devices 20, and executes movement detection ofindividual board surfaces 210 a to 210 d of the whiteboard 200 and otherprocessing.

In order to realize such a function, the image processing device 10includes, for example, a moving image input unit 11, a controlinformation input unit 12, a movement detection unit 13, a writingportion detection unit 14, and an output unit 16 as a functionalconfiguration.

The moving image input unit 11 receives at least one piece of movingimage data output from one or more imaging devices 20 as a video signal,and passes the video signal to the movement detection unit 13 and thewriting portion detection unit 14.

The control information input unit 12 receives control information,which is an instruction or the like from the user input to theinput/output device 30, as a rule signal, and passes the controlinformation to the movement detection unit 13 and the writing portiondetection unit 14.

The movement detection unit 13 receives the moving image data from themoving image input unit 11, detects a straight line portion from a frameof the individual board surfaces 210 a to 210 d, and detects thepresence or absence of movement, the direction and speed of movement,and the like of the individual board surfaces 210 a to 210 d from themotion of the straight line portion. At this time, the movementdetection unit 13 refers to the control information received from thecontrol information input unit 12. The movement detection unit 13 passesa movement detection result of the individual board surfaces 210 a to210 d to the writing portion detection unit 14.

The writing portion detection unit 14 detects the writing portionwritten on the board surfaces 210 a to 210 d by the writer 300 from themoving image data received from the moving image input unit 11. At thistime, the writing portion detection unit 14 executes processingaccording to the movement detection result for each of the boardsurfaces 210 a to 210 d based on the movement detection result of theindividual board surfaces 210 a to 210 d received from the movementdetection unit 13. At this time, the writing portion detection unit 14refers to the control information received from the control informationinput unit 12. The writing portion detection unit 14 passes a detectionresult of the writing portion on the board surfaces 210 a to 210 d tothe output unit 16.

The output unit 16 outputs the detection result of the writing portionby the writing portion detection unit 14 to one or more input/outputdevices 30 and one or more recording devices 40 connected to the imageprocessing device 10. The output unit 16 may directly output the movingimage data imaged by the imaging device 20 to the recording device 40.

The user can check the detection result of the writing portion outputfrom the image processing device 10 with the input/output device 30. Theuser may be able to view the moving image data with the input/outputdevice 30. Further, the user can input the control information from theinput/output device 30. However, the input/output device 30 may not bean independent device, and the image processing device 10 may have thefunction of the input/output device 30.

The control information that can be input by the user includes, forexample, information used for movement determination of the boardsurfaces 210 a to 210 d, the content of processing and control based ona movement determination result, the content of processing on the movingimage data, supplementary information used for movement detection of theboard surfaces 210 a to 210 d, and the like.

The information used for the movement determination of the boardsurfaces 210 a to 210 d includes, for example, information on a straightline portion to be detected from the frame of the board surfaces 210 ato 210 d. The content of processing and control based on the movementdetermination result includes making the content of processing andcontrol different between the board surface 210 whose movement has beendetected and the board surface 210 whose movement has not been detected.

The supplementary information used for movement detection of the boardsurfaces 210 a to 210 d includes, for example, the movement direction ofthe individual board surfaces 210 a to 210 d, and the number,configuration, arrangement, color, and the like of the board surfaces210 a to 210 d.

The storage device 40 records the detection result of the writingportion acquired from the image processing device 10. However, thestorage device 40 may not be an independent device, and the imageprocessing device 10 may have the function of the storage device 40.

FIG. 3 is a block diagram illustrating an example of a hardwareconfiguration of the image processing device 10 according to the firstembodiment of the present disclosure.

As illustrated in FIG. 3 , the image processing device 10 is configuredas a computer that includes, for example, a central processing unit(CPU) 110, a read only memory (ROM) 120, a random access memory (RAM)130, and an in/out (I/O) port 140 as the hardware configuration.

The CPU 110 reads, for example, a program related to image processingstored in the ROM 120, deploys the program on the RAM 130, and operatesaccording to the program. As a result, the CPU 110 can realize variousfunctions as the image processing device 10.

The I/O port 140 is connected to the imaging device 20, the input/outputdevice 30, and the recording device 40, which are external devices ofthe image processing device 10, and transmits and receives various dataand information such as the moving image data, the detection result ofthe writing portion, and the control information to and from theseexternal devices.

Example of Movement Detection Operation of Board Surface

Next, movement detection operation of the board surfaces 210 a to 210 dby the image processing device 10 of the first embodiment will bedescribed with reference to FIG. 4 .

FIG. 4 is a diagram illustrating the movement detection result of theboard surfaces 210 a to 210 d by the image processing device 10according to the first embodiment of the present disclosure, togetherwith the moving image data. FIG. 4 illustrates a frame of moving imagedata at each of times t1, t2, and t3, and a detection result of theboard surfaces 210 a to 210 d in the frame. Here, one frame refers toone shot of a still image constituting the moving image data.

At time t1 in FIG. 4 , the movement detection unit 13 of the imageprocessing device 10 detects the board surfaces 210 a to 210 d bydetecting the straight line portion 261 of the frame of the individualboard surfaces 210 a to 210 d.

The straight line portion 261 of the frame of the board surfaces 210 ato 210 d can be detected, for example, by performing edge extractionbased on a difference in luminance value in the moving image data,binarization of the moving image data, or the like, and performing Houghconversion or the like on the obtained data.

However, in addition to the frame of the board surfaces 210 a to 210 d,for example, straight lines and the like included in the writing portionon the board surfaces 210 a to 210 d can also be detected as thestraight line portion. By referring to the control information from theinput/output device 30, the movement detection unit 13 can determinewhether or not the detected straight line portion is the frame of theboard surfaces 210 a to 210 d based on the length of the straight lineportion 261 of the frame of the individual board surfaces 210 a to 210d, the position where the straight line portion 261 of the frame of theindividual board surfaces 210 a to 210 d is detected, and the like.

For example, in a case where the detected straight line portion is astraight line of the writing portion, it can be determined that thedetected straight portion is not the straight line portion 261 of theframe of the board surfaces 210 a to 210 d, for example, because thedetected straight line portion is shorter than the frame of the boardsurfaces 210 a to 210 d or is detected inside the position of the frameof the board surfaces 210 a to 210 d.

Then, when the straight line portion 261 of the individual boardsurfaces 210 a to 210 d is detected, the movement detection unit 13starts tracking the detected straight line portion 261 of the boardsurfaces 210 a to 210 d from this point on.

At time t2 in FIG. 4 , the board surfaces 210 a and 210 b arrangedvertically are being raised and lowered by the writer 300 or the like.The movement detection unit 13 tracks the straight line portion 261, forexample, by taking a difference from the detection result of thestraight line portion 261 in the previous frame. The movement detectionunit 13 detects that the straight line portion 261 is moving based onthe fact that a difference has arisen in the detection result of thestraight line portion 261 in the current frame from that in the previousframe, and detects that the movement has ended.

At time t3 in FIG. 4 , raising and lowering of the board surfaces 210 aand 210 b ends, and the positions of the board surfaces 210 a and 210 bare replaced with each other, with the board surface 210 a arrangedabove and the board surface 210 b arranged below. The movement detectionunit 13 is continuously tracking the straight line portion 261 of theindividual board surfaces 210 a to 210 d.

Example of Detection Operation of Writing Portion

Next, detection operation of the writing portion on the board surface210 by the image processing device 10 of the first embodiment will bedescribed with reference to FIGS. 5 and 6 . FIG. 5 is a diagramexplaining a method of distinguishing between the writing portion and anon-writing portion by the image processing device 10 according to thefirst embodiment of the present disclosure.

As illustrated in FIG. 5 a , a writing portion 250 written on the boardsurface 210 is recorded in the moving image data. As illustrated in FIG.5 b , for example, the writer 300 or the like standing in front of theboard surface 210 is also recorded in the moving image data such thatthe writer 300 or the like overlaps the board surface 210. The writingportion detection unit 14 of the image processing device 10distinguishes the writing portion 250 from the writer 300 or the likewho appears on the board surface 210 in this manner, and detects thewriting portion 250.

In performing the detection operation of the writing portion 250, thewriting portion detection unit 14 refers to the movement detectionresult of the board surfaces 210 a to 210 d detected by the movementdetection unit 13, and makes the detection operation of the writingportion 250 different between each of the board surfaces 210 a to 210 daccording to the movement detection result.

In a case where the movement detection unit 13 has not detected themovement of a predetermined board surface 210 subject to the detectionoperation of the writing portion 250, the writing portion detection unit14 attempts to detect the writing portion 250 on the board surface 210.

The detection operation of the writing portion 250 by the writingportion detection unit 14 is executed using a reference frame, forexample. The reference frame is updated at each predetermined operationby the writing portion detection unit 14, and is a frame serving as areference of a subsequent operation. For example, the initial referenceframe may be a frame that captures the board surfaces 210 a to 210 dbefore writing is started and before they are replaced with each otherdue to raising and lowering movement.

The writing portion detection unit 14 takes a difference in each pixelbetween the current frame and the reference frame, extracts a pixelhaving a large difference, that is, a pixel having a great change fromthe state in the reference frame, and detects a set portion in whichsuch pixels continuously exist.

Such a set portion can include, for example, the writing portion 250added by the writer 300, the writer 300 performing some action in frontof the board surface 210, and the like. In order to distinguish betweenthem, the writing portion detection unit 14 performs shape evaluation,extraction time evaluation, and stationary time evaluation describedbelow.

First, the writing portion detection unit 14 sets, for each detectedindividual set portion, an outer quadrilateral, which is the smallestquadrilateral including the entire set portion, and an innerquadrilateral, which is the largest quadrilateral that fits into the setportion. In addition, the writing portion detection unit 14 calculatesan evaluation value α of shape from the size of the inner quadrilateralwith respect to the set outer quadrilateral. The evaluation value α ofshape is set such that the smaller the size of the inner quadrilateralis with respect to the outer quadrilateral, the higher the value is.

As illustrated in FIG. 5 a , in a case where the set portion detected bythe writing portion detection unit 14 is the writing portion 250, sincethe writing portion 250 is mainly composed of a combination of linesegments, the inner quadrilateral Pi is relatively smaller than theouter quadrilateral Po. Therefore, the evaluation value α of the shapeis high.

As illustrated in FIG. 5 b , in a case where the set portion detected bythe writing portion detection unit 14 is the writer 300, since thewriter 300 is composed of portions having a collective volume such as ahead and torso, the inner quadrilateral Qi is relatively larger than theouter quadrilateral Qo. Therefore, the evaluation value α of the shapeis low.

Next, the writing portion detection unit 14 checks, for each detectedset portion, the time during which a change has continued to occur inpixels included in the set portion. That is, the writing portiondetection unit 14 calculates a difference between the current frame orthe plurality of frames before the current frame and the reference framefor at least some or all of the pixels included in the set portion.

In addition, the writing portion detection unit 14 accumulates frameshaving a large difference from the reference frame, and sets the numberof consecutive frames among the accumulated frames as the evaluationvalue β of extraction time, which is the time during which a change hasbeen continuously extracted in the pixels included in the set portion.That is, the evaluation value β of extraction time is higher as thenumber of consecutive frames is larger and the extraction time of thechange is longer.

In a case where the set portion detected by the writing portiondetection unit 14 is the writing portion 250, frames having a largedifference continue to be consecutively accumulated every time writingis added, and thus the evaluation value β of extraction time is high.

In a case where the set portion detected by the writing portiondetection unit 14 is the writer 300 or the like moving in front of theboard surface 210, frames having a large difference are no longeraccumulated when the writer 300 has passed through the front of theboard surface 210. Thus, the number of consecutive frames is small andthe evaluation value β of extraction time is low. However, if the writer300 stops moving in front of the board surface 210, the evaluation valueβ of extraction time may become high even if the set portion is thewriter 300.

Next, the writing portion detection unit 14 checks, for each detectedset portion, the time during which there has been no change in pixelsincluded in the set portion. That is, the writing portion detection unit14 calculates a difference between a frame and another frame immediatelybefore the frame among the current frame and the plurality of framesbefore the current frame for at least some or all of the pixels includedin the set portion.

In addition, the writing portion detection unit 14 accumulates frameshaving a large difference from the immediately preceding frame, and setsthe reciprocal of the difference as the evaluation value y of stationarytime, which is the time during which no change has appeared in thepixels included in the set portion. That is, the evaluation value y ofstationary time is higher as the number of frames having a smalldifference from the immediately preceding frame is larger, and thestationary time of each pixel is longer.

In a case where the set portion detected by the writing portiondetection unit 14 is the writing portion 250, since no motion usuallyoccurs in the writing portion written once, the difference from theimmediately preceding frame is 0 or extremely small, and thus theevaluation value y of stationary time is high.

In a case where the set portion detected by the writing portiondetection unit 14 is the writer 300 or the like moving in front of theboard surface 210 or performing an action such as a gesture, thedifference from the immediately preceding frame is large and thus theevaluation value y of stationary time is low.

The writing portion detection unit 14 substitutes each of the evaluationvalues α, β, and γ calculated as described above into an evaluationfunction E (α, β, γ) to obtain a solution, and detects a set portion inwhich the solution of the evaluation function E (α, β, γ) is greaterthan a threshold THr as the writing portion.

Then, the writing portion detection unit 14 determines, for eachdetected writing portion, whether to write and hold each pixel value ofthe writing portion to/at the corresponding pixel position of thecurrent frame or to delete the pixel value without writing the pixelvalue to the corresponding pixel position of the current frame.

For example, in a case where the pixel position of the writing portiondetected in the current frame coincides with the pixel position held asthe writing portion in the previous frame, the writing portion detectionunit 14 deletes each pixel value of the writing portion without writingthe pixel value to the corresponding pixel position of the currentframe. The reason is as follows.

In order to be detected as the writing portion in a predetermined frame,the solution of the evaluation function E (α, β, γ) needs to be greaterthan the threshold THr, and such a state means that a great change hasoccurred at the pixel position. If such a great change occurs again atthe pixel position previously held as the writing portion in the currentframe, it is considered that such a state means that the previouslywritten portion has been erased by the writer 300 or the like in thecurrent frame.

For the above reason, the writing portion detection unit 14 deletes thewriting portion detected at the same position as the pixel positionpreviously detected as the writing portion from the current frame.

For other writing portions detected in the current frame, the writingportion detection unit 14 writes and holds each pixel value of thewriting portion in the current frame. Alternatively, the writing portiondetection unit 14 may hold only information of the pixel position ofeach pixel included in the writing portion, instead of writing the pixelvalue of the writing portion to the corresponding pixel position of thecurrent frame.

The writing portion detection unit 14 sets a current frame for which theholding result and the deletion result of each pixel value of thewriting portion have been reflected as a new reference frame. Suchupdating of the reference frame is performed, for example, for each ofthe board surfaces 210 a to 210 d.

In the case of a type in which the plurality of board surfaces 210 a to210 d are simultaneously moved and replaced as in the whiteboard 200 ofthe embodiment, the reference frame may be updated for each of pairs ofboard surfaces that are moved at a time, that is, a pair of the boardsurfaces 210 a and 210 b or a pair of the board surfaces 210 c and 210d.

The writing portion detection unit 14 uses the newly updated referenceframe when performing detection of the writing portion for the nextframe to the current frame.

With such processing, the writing portion detection unit 14 can performdifferent processing according to the movement determination result ofthe board surfaces 210 a to 210 d. A specific example thereof isillustrated in FIG. 6 .

FIG. 6 is a diagram illustrating the detection result of the writingportion by the image processing device 10 according to the firstembodiment of the present disclosure, together with the moving imagedata. FIG. 6 illustrates a frame of the moving image data and thedetection result of the writing portion in the frame at each of timest1, t2, and t3. Here, the moving image data at times t1, t2, and t3 isthe same as the moving image data at times t1, t2, and t3 in FIG. 4described above.

At time t1 in FIG. 6 , the writing portion detection unit 14 detects thewriting portion 250 for each of the board surfaces 210 a to 210 d. Attime t1, no movement occurs on any of the board surfaces 210 a to 210 d.In this case, the writing portion detection unit 14 detects the writingportion 250 for all the board surfaces 210 a to 210 d.

At time t2 in FIG. 6 , movement occurs in the board surfaces 210 a and210 b vertically arranged. In this case, the writing portion detectionunit 14 makes detection processing of the writing portion 250 differentaccording to the movement detection result of the individual boardsurfaces 210 a to 210 d.

More specifically, the writing portion detection unit 14 detects thewriting portion 250 in a region corresponding to the board surfaces 210c and 210 d whose movement has not been detected by the movementdetection unit 13 in the frame of the moving image data at time t2 tohold and delete each pixel value of the detected writing portion 250 asnecessary.

On the other hand, the writing portion detection unit 14 does not detectthe writing portion 250 in a region corresponding to the board surfaces210 a and 210 b whose movement has been detected by the movementdetection unit 13 in the frame of the moving image data at time t2. Inaddition, the writing portion detection unit 14 deletes each pixel valueheld as the writing portion 250 of the board surfaces 210 a and 210 b inthe previous frame from the current frame. Furthermore, the writingportion detection unit 14 deletes the reference frame of the boardsurfaces 210 a and 210 b at that time.

As described above, if the movement occurs in any of the board surfaces210, each pixel included in the region greatly changes. For this reason,if a difference in the frame after the movement of the board surface 210occurs is taken by referring to the reference frame before the movementof the board surface 210 occurs, a large difference is detected even ina portion other than the writing portion 250, which makes it difficultto correctly separate the writing portion 250 and the non-writingportion. By deleting the reference frame corresponding to the boardsurface 210 at the timing when the movement of the board surface 210 hasbeen detected, the above-described erroneous detection can besuppressed.

Through this processing, as the detection result of the writing portion250 at time t2, a state is obtained in which the left regioncorresponding to the board surfaces 210 a and 210 b does not have thewriting portion 250, while the right region corresponding to the boardsurfaces 210 c and 210 d has the writing portion 250. Such a state asthe detection result of the writing portion 250 is maintained until themovement of the board surfaces 210 a and 210 b ends.

When raising and lowering of the board surfaces 210 a and 210 b ends attime t3 in FIG. 6 , the writing portion detection unit 14 starts thedetection of the writing portion 250 for each of the board surfaces 210a to 210 d again. In addition, the writing portion detection unit 14newly registers the frame at that time as the reference frame.

Processing Example of Image Processing Device

Next, an example of image processing by the image processing device 10of the first embodiment will be described with reference to FIG. 7 .FIG. 7 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device 10 according to the firstembodiment of the present disclosure.

In the flowchart in FIG. 7 , one cycle corresponds to processing for onepredetermined frame constituting the moving image data, and when theprocessing for one frame is completed, cycles for subsequent frames aresequentially repeated.

As described above, in performing image processing, the image processingdevice 10 registers, as the initial reference frame, a reference framethat captures, for example, the board surfaces 210 a to 210 d beforewriting is started and before they are replaced with each other due tothe raising and lowering movement. In other words, this processing canalso be regarded as processing in Step S114 in FIG. 7 .

That is, since the movement detection of the board surfaces 210 a to 210d by the movement detection unit 13 (Step S101) does not detect themovement of any of the board surfaces 210 a to 210 d (Step S102: No),and there is no reference frame (Step S106: No) because of the initialstage of processing by the image processing device 10, the writingportion detection unit 14 registers the frame of the moving image dataat that time as the reference frame (Step S114).

Then, processing returns to the beginning again, for example, in thenext frame, the movement detection unit 13 performs the movementdetection of the board surfaces 210 a to 210 d (Step S101) by detectingand tracking the straight line portion of the board surfaces 210 a to210 d, and the writing portion detection unit 14 refers to whether ornot the movement detection unit 13 has detected the movement of any ofthe board surfaces 210 a to 210 d (Step S102).

In the initial stage of processing by the image processing device 10, itis considered that the writer 300 or the like does not move the boardsurfaces 210 a to 210 d because nothing has been written on the boardsurfaces 210 a and 210 c arranged below or there is still a sufficientspace to add writing,

Thus, for example, in a case where the movement detection unit 13 hasnot detected the movement of any of the board surfaces 210 a to 210 d(Step S102: No), the writing portion detection unit 14 determineswhether or not there is an already registered or updated reference frame(Step S106).

In a case where there is the reference frame (Step S106: Yes), thewriting portion detection unit 14 extracts a set portion in which pixelshaving a large difference between the reference frame and the currentframe concentrate (Step S107). At this time, the extracted set portionmay include, in addition to the writing portion, the non-writing portionsuch as the writer 300 or the like performing some action.

The writing portion detection unit 14 calculates the evaluation value αof shape by comparing the outer quadrilateral and the innerquadrilateral for each set portion (Step S108). The evaluation value αof shape tends to be higher in the writing portion than in thenon-writing portion such as the writer 300 extracted as the set portion,for example.

For each set portion, the writing portion detection unit 14 calculatesthe evaluation value β of extraction time from the number of consecutiveframes having a large difference from the reference frame among thecurrent frame and the plurality of frames before the current frame (StepS109). The extraction time at this time indicates the time during whicha change has been continuously extracted in a predetermined set portion,and the evaluation value β of extraction time tends to be higher in thewriting portion than in the non-writing portion.

For each set portion, the writing portion detection unit 14 calculatesthe evaluation value γ of stationary time from the reciprocal of thedifference of a frame having a large difference from another frameimmediately before the frame among the current frame and the pluralityof frames before the current frame (Step S110). The stationary time atthis time indicates the time during which no change has appeared in thepixels included in the set portion, and the evaluation value γ ofstationary time tends to be higher in the writing portion than in thenon-writing portion.

The writing portion detection unit 14 obtains a solution of theevaluation function E (α, β, γ) for each set portion, and determines aset portion in which the solution of the evaluation function E (α, β, γ)is greater than the threshold THr as the writing portion (Step S111).

The writing portion detection unit 14 determines whether to hold ordelete the individual set portions determined to be the writing portion(Step S112). For example, in a case where the pixel position of thewriting portion detected in the frame subject to the current processingcoincides with the pixel position of the writing portion held in theprevious frame, the writing portion detection unit 14 deletes the setportion without holding it, determining that the current detectionresult indicates that the previously written writing portion has beenerased by the writer 300.

The writing portion detection unit 14 reflects the holding and deletionresults for the individual set portions determined to be the writingportion in the frame subject to the current processing, and use theframe as a new reference frame to update the reference frame (Step S113).

As described above, since writing is performed exclusively on the boardsurfaces 210 a and 210 c arranged below for a while after the start ofprocessing by the image processing device 10, it is considered that themovement of the board surfaces 210 a to 210 d does not occur, andprocessing in Steps S106 to S113 is repeated while the reference frameis sequentially updated.

Then, the lower board surfaces 210 a and 210 c are filled with thewriting portions, and one of the board surfaces 210 a to 210 d isreplaced at a predetermined timing.

Thus, for example, when the movement detection unit 13 detects themovement of any of the board surfaces 210 a to 210 d (Step S102: Yes),the writing portion detection unit 14 branches processing between theboard surface 210 whose movement has occurred and the board surface 210whose movement has not occurred (Step S103).

That is, in a case where the region to be processed from now on in apredetermined frame is the region corresponding to the board surface 210where the current movement has occurred (Step S103: Yes), the writingportion detection unit 14 deletes the writing portion held in the regioncorresponding to the board surface 210 where the current movement hasoccurred in the previous frame (Step S104).

The writing portion detection unit 14 deletes the reference frame forthe board surface 210 where the current movement has occurred (StepS105).

On the other hand, in a case where the region to be processed from nowon in the predetermined frame is not the region corresponding to theboard surface 210 where the current movement has occurred (Step S103:No), the writing portion detection unit 14 performs processing in StepsS106 to S113 as described above.

As described above, the writing portion detection unit 14 performsbranched processing in Steps S103 to S113, so that the data up to thattime is once reset in the region corresponding to the board surface 210whose movement has occurred, and the data up to that time is maintainedin the region corresponding to the board surface 210 whose movement hasnot occurred.

After that, when processing returns to the beginning again, for example,in the next frame, in a case where the movement has not occurred in anyof the board surfaces 210 a to 210 d (Step S102: No), or in a case whereno movement has occurred in the board surface 210 where the currentmovement has occurred (Step S103: No), the reference frame does notexist for the board surface 210 where the current movement has occurred(Step S105: No), and therefore the reference frame is registered againfor the board surface 210 where the current movement has occurred (StepS114).

Then, image processing by the image processing device 10 of the firstembodiment ends.

Summary

In recent years, opportunities for recording lectures, training,seminars, and conferences, or the like are increasing in various scenesincluding education sites.

In such recording, a lecturer, a teacher, a presenter, or the likegiving explanations by actually writing on a whiteboard, a blackboard,an electronic blackboard, or the like for the moment is often recorded,instead of giving explanations using presentation materials or the like,which is easily distributed as digital data.

For this reason, techniques for performing various image analysis andimage processing on a recorded video including the whiteboard, theblackboard, the electronic blackboard, or the like have been proposed.

According to the image processing device 10 of the first embodiment, ifthe movement of the board surface 210 has been detected when separatingthe writing portion and the non-writing portion based on a changebetween the frames of the moving image data, processing corresponding tothe movement of the board surface 210 is executed. As a result, imageprocessing can be performed with high accuracy even in a case wherewriting is performed on the movable board surface 210.

According to the image processing device 10 of the first embodiment, ifthe movement of the board surface 210 has been detected, the detectionresult of the writing portion on the board surface 210 is deleted. As aresult, the spread of the change between the frames to the entire boardsurface 210 or the entire screen of the moving image data due to themovement of the board surface 210 is suppressed, making it possible toaccurately distinguish between the writing portion and the non-writingportion.

According to the image processing device 10 of the first embodiment, ifthe movement of the board surface 210 has been detected, the detectionresult of the writing portion on the board surface 210 whose movementhas been detected is deleted, and the detection result of the writingportion on the board surface 210 whose movement has not been detected isheld. As a result, it is possible to more accurately distinguish betweenthe change between the frames due to the movement of the board surface210 and the change between the frames due to the addition of writing orthe like.

According to the image processing device 10 of the first embodiment, themovement of the board surface 210 is detected by detecting the straightline portion of the frame of the board surface 210 and detecting themovement of the straight line portion. As a result, the movement of theboard surface 210 can be easily detected with high accuracy withoutrequiring enormous data analysis or the like, for example. In addition,since the board surface 210 itself is directly detected, the movement ofthe board surface 210 can be detected, for example, even in the case ofthe board surface 210 on which writing is not performed.

According to the image processing device 10 of the first embodiment, forexample, the straight line included in the writing portion and the frameof the board surface 210 are distinguished based on the length of thestraight line portion. The length of the straight line portion of theframe of the board surface 210 can be determined based on the controlinformation input from the input/output device 30 and the supplementaryinformation including information such as the direction, number,configuration, arrangement, and color of the movement of the boardsurface 210. As a result, erroneous detection can be suppressed.

According to the image processing device 10 of the first embodiment, thedetection result of the writing portion extracted with high accuracy asdescribed above can be used for various purposes.

In the first embodiment described above, the whiteboard 200 includes theplurality of board surfaces 210 a and 210 c. However, a whiteboard orthe like on which the writer performs writing may include only one boardsurface.

In the first embodiment described above, the whiteboard 200 is fixed to,for example, a wall surface of a building or the like. However, thewhiteboard or the like on which the writer performs writing may be acaster-equipped whiteboard or the like that allows itself to be freelymoved.

In the first embodiment described above, a blackboard, an electronicblackboard, or the like is used in addition to the whiteboard 200.However, a sheet-like paper medium such as poster paper attached to thewhiteboard 200, the wall surface of a building, the blackboard, or thelike may be used as a writing target.

In this case, the paper medium may be attached to the caster-equippedwhiteboard or the like for a movable configuration. Alternatively, aplurality of paper media may be stacked and attached, and may beconfigured so as to be turned one by one every time writing on one papermedium is completed. Even in the case of such a configuration, in thefirst embodiment described above, it is possible to continue tocorrectly detect the writing portion on the paper medium in response toa great change that occurs in the video every time the paper mediumfilled with the writing portion is turned and the next new paper mediumappears.

In the first embodiment described above, the detection result of thewriting portion according to the movement of the board surface 210 isoutput to the recording device 40. However, the image processing devicemay output the moving image data received from the imaging device andthe movement detection result of the board surface 210 to the recordingdevice in association with each other.

First Modification

Next, an image processing device of a first modification of the firstembodiment will be described with reference to FIG. 8 . In the imageprocessing device of the first modification, a method of detecting themovement of the board surfaces 210 a to 210 d is different from that ofthe first embodiment described above.

A movement detection unit of the image processing device of the firstmodification detects the movement of the board surfaces 210 a to 210 dby estimating the optical flow of the entire screen in a predeterminedframe constituting the moving image data. In optical flow, for example,the motion of an object detected based on the luminance gradient isrepresented by a vector, and the motion of the object is estimated fromthe direction and size of the vector.

In a case where the movement of a predetermined board surface 210 isdetected using optical flow, the writing portion that moves with themovement of the predetermined board surface 210 is detected based on theluminance gradient. Since the movement of the board surface 210 isusually performed after the board surface 210 is filled with the writingportion to some extent, the movement of the board surface 210 can bedetected also by detecting the movement of the writing portion on theboard surface 210 in this manner.

However, at this time, the writer 300 performing some action in front ofthe board surface 210 can also be detected. Therefore, the movementdetection unit of the first modification groups the optical flowsrepresented by the vectors into each vector group in which apredetermined number of vectors are concentrated. In addition, themovement detection unit of the first modification determines whether ornot the detected movement is global from the size of the grouped vectorgroup.

For example, in a case where the grouped vector group is a writingportion over substantially the entire board surface 210, the vectorgroup has a predetermined collective size, and thus the detectedmovement should be global.

On the other hand, in a case where the grouped vector group is thewriter 300 or the like standing in front of the board surface 210, thevector group is small and thus the detected movement should be local.

From the above, the movement detection unit of the first modificationcan identify the moving board surface 210 and the writer 300 performingsome action, and can correctly detect the movement of the board surface210.

In a case where the plurality of board surfaces 210 simultaneously movedue to raising and lowering as in the whiteboard 200 described above, orin a case where the plurality of board surfaces simultaneously move inthe horizontal direction due to lateral sliding, the movement detectionunit of the first modification may detect the movement of the boardsurfaces 210 based on the detection of the global optical flow havingthe opposite vector.

FIG. 8 is a diagram illustrating a movement detection result of theboard surfaces 210 a to 210 d by the image processing device accordingto the first modification of the first embodiment of the presentdisclosure, together with the moving image data.

FIG. 8 illustrates a frame of the moving image data and the movementdetection result of the board surfaces 210 a to 210 d in the frame ateach of times t1, t2, and t3, together with a detection result of thewriting portion for reference. The moving image data at times t1, t2,and t3 is the same as the moving image data at times t1, t2, and t3 inFIGS. 4 and 6 described above.

At time t1 in FIG. 8 , writing is performed on the board surfaces 210 ato 210 d, but no movement occurs on any of the board surfaces 210 a to210 d; thus, the movement detection unit of the first modification doesnot detect the movement of the board surfaces 210 a to 210 d usingoptical flow.

Since the movement of the board surfaces 210 a to 210 d is not detected,a writing portion detection unit of the first modification detects thewriting portions 250 of all the board surfaces 210 a to 210 d. Then, thedetection result of the writing portion 250 at time t1 is obtained.

At time t2 in FIG. 8 , when the board surfaces 210 a and 210 bvertically arranged are moving due to raising and lowering, the movementdetection unit of the first modification detects the optical flow 262having a vector in the vertically opposite direction from the actualmovement of the writing portion recorded in the moving image data, anddetermines that the movement of the board surfaces 210 a and 210 b hasoccurred because the optical flow is global.

Since the movement of the board surfaces 210 a and 210 b has beendetected, the writing portion detection unit of the first modificationdeletes the detection result of the writing portion 250 of the boardsurfaces 210 a and 210 b, and holds the detection result of the writingportion 250 of the board surfaces 210 c and 210 d that remainstationary. Then, the detection result of the writing portion 20 at timet2 is obtained.

At time t3 in FIG. 8 , the moving image data includes the board surfaces210 a to 210 d all of which are in the stationary state again, and thewriter 300 moving in front of the board surfaces 210 a to 210 d.Therefore, the movement detection unit of the first modification detectsthe optical flow 262 s due to the movement of the writer 300.

However, the optical flow 262 s is not global, and the direction ofmovement is the lateral direction, which is different from the directionof the board surfaces 210 a to 210 d that move vertically. Therefore,the movement detection unit of the first modification determines thatthe optical flow 262 s is not the movement of the board surfaces 210 aand 210 b.

Since the movement of the board surfaces 210 a and 210 b ends at timet3, the writing portion detection unit of the first modification detectsthe writing portions 250 of all the board surfaces 210 a to 210 d again.Then, the detection result of the writing portion 250 at time t3 isobtained.

As described above, the same detection result of the writing portion 250as in the case of the above-described first embodiment can be obtainedalso by the movement detection of the board surfaces 210 a to 210 dusing optical flow.

According to the image processing device of the first modification, themovement of the board surface 210 is detected by estimating the opticalflow for the movement of the writing portion associated with themovement of the board surface 210. As a result, the movement of theboard surface 210 can be easily detected without requiring enormous dataanalysis or the like, for example. In addition, using the optical flowmakes it possible to detect even fine motion and slow motion of lessthan one pixel, for example. Furthermore, the movement of the boardsurface can be detected even in a case where the frame of the boardsurface does not appear as a clear edge on the image and in a case wherethe board surface is curved.

According to the image processing device of the first modification, theoptical flows are grouped and a distinct is made between the movement ofthe writer 300 and the movement of the board surface 210 based on thesize and the movement direction of the grouped region. The size, themovement direction, and the like of the board surface 210 can bedetermined based on the control information input from the input/outputdevice 30 and the supplementary information including information suchas the direction, number, configuration, arrangement, and color of themovement of the board surface 210. As a result, erroneous detection canbe suppressed.

Second Modification

Next, an image processing device of a second modification of the firstembodiment will be described with reference to FIG. 9 . In the imageprocessing device of the second modification, a method of detecting themovement of the board surfaces 210 a to 210 d is different from that ofthe first embodiment described above.

A movement detection unit of the image processing device of the secondmodification detects the movement of the board surfaces 210 a to 210 dusing general object recognition in a predetermined frame constitutingthe moving image data. In general object recognition, for example, anobject included in a video is recognized by a general name, and themotion of the object is detected by tracking a position of therecognized object. In order to cause an object to be recognized by ageneral name, for example, a method such as machine learning is used.

In a case where the movement of the board surfaces 210 a to 210 d isdetected using general object recognition, the board surfaces 210 a to210 d that can be individually moved are each recognized by a generalname such as a “whiteboard”, and the positions thereof are tracked basedon a change between the frames of the positions. However, the movementdetection unit of the second modification may detect the movement of the“whiteboard” by tracking not a position but the center of gravity of theobject recognized as the “whiteboard”, that is, the center of gravity ofthe board surfaces 210 a to 210 d.

At this time, it is preferable to learn an object that can be includedin the moving image data in addition to the board surfaces 210 a to 210d, for example, by causing the writer 300 or the like to be recognizedby a general name such as a “person”.

FIG. 9 is a diagram illustrating a movement detection result of theboard surfaces 210 a to 210 d by the image processing device accordingto the second modification of the first embodiment of the presentdisclosure, together with the moving image data.

FIG. 9 illustrates a frame of the moving image data and the movementdetection result of the board surfaces 210 a to 210 d in the frame ateach of times t1, t2, and t3, together with a detection result of thewriting portion for reference. The moving image data at times t1, t2,and t3 is the same as the moving image data at times t1, t2, and t3 inFIGS. 4 and 6 described above.

At time t1 in FIG. 9 , the movement detection unit of the secondmodification recognizes each of the board surfaces 210 a to 210 dincluded in the moving image data as a “whiteboard” 263, and similarlyrecognizes the writer 300 included in the moving image data as a“person” 263 s. Thereafter, the movement detection unit of the secondmodification starts tracking the positions of the “whiteboard” 263 andthe “person” 263 s.

Since the movement of the board surfaces 210 a to 210 d recognized asthe “whiteboard” 263 is not detected at time t1, a writing portiondetection unit of the second modification detects the writing portions250 of all the board surfaces 210 a to 210 d. Then, the detection resultof the writing portion 250 at time t1 is obtained.

When the board surfaces 210 a and 210 b vertically arranged are movingdue to raising and lowering at time t2 in FIG. 9 , the movementdetection unit of the second modification determines that the movementof the board surfaces 210 a and 210 b has occurred according to a changein the position of the “whiteboard” 263 corresponding to the boardsurfaces 210 a and 210 b.

Since the movement of the board surfaces 210 a and 210 b has beendetected, the writing portion detection unit of the second modificationdeletes the detection result of the writing portion 250 of the boardsurfaces 210 a and 210 b, and holds the detection result of the writingportion 250 of the board surfaces 210 c and 210 d that remainstationary. Then, the detection result of the writing portion 250 attime t2 is obtained.

At time t3 in FIG. 9 , the moving image data includes the board surfaces210 a to 210 d all of which are in the stationary state again, and thewriter 300 moving in front of the board surfaces 210 a to 210 d.Therefore, the movement detection unit of the second modificationdetermines that the movement of the “person” 263 s, who is the writer300, has occurred and the movement of the “whiteboard” 263, which is theboard surfaces 210 a to 210 d, has ended.

Since the movement of the board surfaces 210 a and 210 b has ended, thewriting portion detection unit of the second modification detects thewriting portions 250 of all the board surfaces 210 a to 210 d again.Then, the detection result of the writing portion 250 at time t3 isobtained.

As described above, the same detection result of the writing portion 250as in the case of the above-described first embodiment can be obtainedalso by the movement detection of the board surfaces 210 a to 210 dusing general object recognition.

According to the image processing device of the second modification, theboard surfaces 210 a to 210 d are recognized by general objectrecognition, and the movement of the board surfaces 210 a to 210 d isdetected by tracking the position or the center of gravity of the boardsurfaces 210 a to 210 d. As a result, the movement of the board surfaces210 a to 210 d can be detected with high accuracy.

According to the image processing device of the second modification, adistinct is made between the movement of the writer 300 and the movementof the board surfaces 210 a to 210 d by recognizing the writer 300 aswell using general object recognition. As a result, erroneous detectioncan be suppressed.

Third Modification

Next, an image processing device of a third modification of the firstembodiment will be described. The image processing device of the thirdmodification is different from the first embodiment described above inthat the image processing device has a function of tracking the movingplate surface.

A movement detection unit of the image processing device of the thirdmodification detects movement of a predetermined board surface usingstraight line detection of the frame of the board surface, optical flowdetection of the writing portion on the board surface, general objectrecognition of the board surface, or the like, as described above.

In addition, the movement detection unit of the third modificationchanges the angle of view of the imaging device 20 using a controlsignal for controlling the imaging device 20 in a case where the boardsurface whose movement has been detected continues to move out of theframe of the moving image data.

More specifically, the movement detection unit of the third modificationcauses the imaging device 20 to perform imaging at a wider angle ofview. Such control is possible in a case where the imaging device 20has, for example, an optical or electronic zoom-up function and azoom-back function. As described above, widening the angle of view ofthe imaging device 20 enables a wider range of imaging, and enables theboard surface that has moved out of the frame of the moving image datato be captured in the frame again.

Alternatively, the movement detection unit of the third modificationcauses the imaging device 20 to move the angle of view in at least oneof the horizontal direction and the vertical direction. Such control ispossible in a case where the imaging device 20 has, for example, afunction of mechanically panning and tilting. The direction of themovement (pan/tilt) of the angle of view can be determined based on themovement direction of the board surface detected by the movementdetection unit of the third modification immediately before the boardsurface moves out of the frame. As described above, moving the angle ofview of the imaging device 20 enables the angle of view to be directedin the direction of the movement of the board surface, and enables theboard surface that has moved out of the frame of the moving image datato be captured in the frame again.

Alternatively, the movement detection unit of the third modificationcauses the imaging device 20 to perform imaging at a wider angle of viewwhile moving the angle of view in at least one of the horizontaldirection and the vertical direction. This enables the board surfacethat has moved out of the frame of the moving image data to be capturedin the frame again, and enables the board surface to be imaged at a moreappropriate angle of view.

The configuration of the movement detection unit of the thirdmodification is suitably applied to, for example, a case where itbecomes difficult to capture the entire whiteboard 200 within the framebecause the writing portion on a predetermined board surface 210 of theabove-described whiteboard 200 is enlarged for display.

Furthermore, the configuration of the movement detection unit of thethird modification is suitably applied to a case where the direction andrange of the movement are irregular and relatively wide, for example, asin a caster-equipped whiteboard or the like.

According to the image processing device of the third modification, theangle of view of the imaging device 20 is changed in a case where theboard surface has moved out of the frame of the moving image data. As aresult, it is possible to continue to capture the board surface withinthe frame of the moving image data, and continue the detection of thewriting portion on the board surface.

Second Embodiment

A second embodiment will be described with reference to the drawings. Inthe second embodiment, an image processing device having a function ofsuperimposing a detected writing portion on moving image data imaged byan imaging device 20 will be described as an example of the use of thedetection result of the writing portion obtained as in the firstembodiment or the first to third modifications described above. Suchsuperimposition processing improves the visibility of the writingportion.

Configuration Example of Image Processing Device

FIG. 10 is a block diagram illustrating an example of a functionalconfiguration of an image processing device 10 a according to the secondembodiment of the present disclosure.

As illustrated in FIG. 10 , the image processing device 10 a isconnected to one or more imaging devices 20, input/output devices 30,and recording devices 40, and constitutes a part of an image processingsystem 2 of the second embodiment. The image processing device 10 a ofthe second embodiment is also a computer including, for example, a CPU,a ROM, a RAM, and an I/O port.

In addition, the image processing device 10 of the second embodimentincludes a data generation unit 15 a in addition to a moving image inputunit 11, a control information input unit 12, a movement detection unit13, a writing portion detection unit 14, and an output unit 16, whichare the same functional configuration as that of the first embodimentdescribed above.

However, the image processing device 10 a may include, instead of themovement detection unit 13 that detects the movement of the boardsurfaces 210 a to 210 d using straight line detection, the movementdetection unit of the first modification or the second modification ofthe first embodiment described above that detects the movement of theboard surfaces 210 a to 210 d using optical flow, general objectrecognition, or the like.

The data generation unit 15 a generates the writing data based on adetection result of the writing portion acquired from the writingportion detection unit 14. In addition, the data generation unit 15 asuperimposes the generated writing data on the moving image datareceived from the moving image input unit 11 to generate compositemoving image data. Through the series of processing, the data generationunit 15 a refers to control information received from the controlinformation input unit 12.

The output unit 16 outputs the composite moving image data generated bythe data generation unit 15 a to one or more input/output devices 30 andone or more recording devices 40 connected to the image processingdevice 10 a. As a result, the user can view the composite moving imagedata with the input/output device 30.

Example of Data Generation Operation

Next, generation operation of the writing data and the composite movingimage data by the image processing device 10 a according to the secondembodiment will be described with reference to FIG. 11 .

FIG. 11 is a diagram illustrating a generation result of the writingdata and the composite moving image data by the image processing device10 a according to the second embodiment of the present disclosure,together with the moving image data. FIG. 11 illustrates a frame of themoving image data, and the generation result of the writing data and thegeneration result of the composite moving image data in the frame ateach of times t1, t2, and t3.

The moving image data at times t1, t2, and t3 is the same as the movingimage data at times t1, t2, and t3 in FIGS. 4 and 6 of the firstembodiment described above, and it is thus assumed that the samemovement detection result of the board surfaces 210 a to 210 d and thesame generation result of the writing data as those in FIGS. 4 and 6 ofthe first embodiment described above are obtained.

At time t1 in FIG. 11 , the data generation unit 15 a of the imageprocessing device 10 a extracts the detection result of the writingportion 250 by the writing portion detection unit 14 and generates thewriting data. Since no movement has occurred on any of the boardsurfaces 210 a to 210 d at time t1, the data generation unit 15 agenerates the writing data including the writing portion 250 on all theboard surfaces 210 a to 210 d. In addition, the data generation unit 15a superimposes the generated writing data on the moving image data attime t1 to generate composite moving image data.

The moving image data at time t1 includes the board surfaces 210 a to210 d in a stationary state and a writer 300 standing in front of theboard surfaces 210 a to 210 d. In addition, the writing data at time t1includes the writing portions 250 written on the individual boardsurfaces 210 a to 210 d as described above. The writing data issuperimposed on the moving image data so as to overlap the actualwriting portion appearing in the moving image data.

As a result, in the composite moving image data at time t1, the writingportion hidden by the writer 300 is superimposed on the video of thewriter 300 while the positional relationship between the writer 300 andthe writing portion is maintained, thus improving the visibility of thewriting portion 250.

Similarly at time t2 in FIG. 11 , the data generation unit 15 a extractsthe detection result of the writing portion 250 by the writing portiondetection unit 14 to generate the writing data.

At time t2, movement occurs in the board surfaces 210 a and 210 bvertically arranged. Therefore, the data generation unit 15 a extractsdata of each pixel value of the writing portion 250 held by the writingportion detection unit 14 from the region corresponding to the boardsurfaces 210 c and 210 d whose movement has not been detected, andreflects the data in the writing data to be generated.

In addition, since the writing portion 250 is deleted by the writingportion detection unit 14 from the region corresponding to the boardsurfaces 210 a and 210 b where the movement is detected, the datageneration unit 15 a reflects the state as it is in the writing data.

As a result, in the writing data at time t2, the left regioncorresponding to the board surfaces 210 a and 210 b does not have thewriting portion 250, while the right region corresponding to the boardsurfaces 210 c and 210 d has the writing portion 250. Such a state ofthe writing data is maintained until the movement of the board surfaces210 a and 210 b ends.

The data generation unit 15 a superimposes the writing data generated asdescribed above on the moving image data at time t2 to generatecomposite moving image data.

The moving image data at time t2 includes the board surfaces 210 a and210 b that are moving, the board surfaces 210 c and 210 d that remainstationary, and the writer 300 standing in front of the board surfaces210 a to 210 d. In addition, the writing data at time t2 does notinclude the writing portion written on the board surfaces 210 a and 210b that are moving, and includes only the writing portion 250 written onthe board surfaces 210 c and 210 d, as described above.

Therefore, when the writing data at time t2 is superimposed on themoving image data at time t2, in the composite moving image data at timet2, the writing portion is not superimposed on the board surfaces 210 aand 210 b, and the writing portion appearing in the moving image data isdisplayed as it is. Therefore, the writing portion behind the writer 300standing in front of the board surfaces 210 a and 210 b remains hiddenby the writer 300.

On the other hand, the writing portion 250 included in the writing datais superimposed on the video of the writer 300 on the board surfaces 210c and 210 d.

At time t3 in FIG. 11 as well, the data generation unit 15 a extractsthe detection result of the writing portion 250 by the writing portiondetection unit 14 to generate the writing data. Since raising andlowering of the board surfaces 210 a and 210 b ends at time t3, the datageneration unit 15 a generates the writing data including the writingportions 250 of all the board surfaces 210 a to 210 d again. Inaddition, the data generation unit 15 a superimposes the generatedwriting data on the moving image data at time t3 to generate compositemoving image data.

The moving image data at time t3 includes the board surfaces 210 a to210 d all of which are in the stationary state again, and the writer 300standing in front of the board surfaces 210 a to 210 d. In addition, thewriting data at time t3 includes the writing portions 250 on all theboard surfaces 210 a to 210 d as described above.

Therefore, when the writing data at time t3 is superimposed on themoving image data at time t3, the writing portion 250 is superimposed onall the board surfaces 210 a to 210 d in the composite moving image dataat time t3. As a result, the writing portion hidden by the writer 300 issuperimposed on the video of the writer 300, thus improving thevisibility of the writing portion 250.

Processing Example of Image Processing Device

Next, an example of image processing by the image processing device 10 aof the second embodiment will be described with reference to FIG. 12 .FIG. 12 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device 10 a according to the secondembodiment of the present disclosure.

In the flowchart in FIG. 12 , processing in Steps S101 to S114 is thesame as processing in Steps S101 to S114 in FIG. 8 of the firstembodiment described above.

That is, in a case where the movement detection of the board surfaces210 a to 210 d by the movement detection unit 13 (Step S101) does notdetect the movement of all or some of the board surfaces 210 (Step S102or Step S103: No), and there is no reference frame (Step S106: No)before the registration of the first reference frame in the initialstage of processing by the image processing device 10 a or immediatelyafter the reference frame is deleted because the movement of the boardsurfaces 210 a to 210 d occurs in the previous frame, the writingportion detection unit 14 registers the frame of the moving image dataat that time as the reference frame (Step S114). Then, processingreturns to the beginning again, for example, in the next frame.

In a case where there is the reference frame (Step S106: Yes) in theabove case (Step S102 or Step S103: No), the writing portion detectionunit 14 detects the writing portion, and reflects the holding anddeleting results for the individual set portions determined to be thewriting portion in the frame to be processed to update the referenceframe (Steps S107 to S113).

The data generation unit 15 a generates the writing data based onprocessing in Steps S107 to S113 by the writing portion detection unit14 (Step S115).

The data generation unit 15 a superimposes the generated writing data onthe frame of the moving image data subject to the current processing(Step S116). Then, processing returns to the beginning again, forexample, in the next frame.

In a case where the movement of any of the board surfaces 210 a to 210 dhas been detected (Step S102: Yes), and the region to be processed fromnow on is the region corresponding to the board surface 210 where thecurrent movement has occurred (Step S103: Yes), the writing portiondetection unit 14 deletes the writing portion held in the regioncorresponding to the board surface 210 where the current movement hasoccurred in the previous frame and the reference frame of the boardsurface 210 where the current movement has occurred (Steps S104 toS105).

Based on processing in Steps S104 to S105 and processing in Steps S107to S113 by the writing portion detection unit 14, the composite movingimage generation unit 15 generates the writing data not having thewriting portion in the region corresponding to the board surface 210whose movement has occurred and having the writing portion in the regioncorresponding to the board surface 210 whose movement has not occurred(Step S115) .

The composite moving image generation unit 15 superimposes the generatedwriting data on the frame of the moving image data subject to thecurrent processing (Step S116) .

Then, image processing by the image processing device 10 a of the secondembodiment ends.

Summary

According to the image processing device 10 a of the second embodiment,the writing data is generated based on the detection result of thewriting portion, and the composite moving image data is generated bysuperimposing the writing data on the moving image data. As a result, itis possible to enhance the visibility of the writing portion in a casewhere a recorded content such as lecture notes and meeting minutes areviewed.

Third Embodiment

A third embodiment will be described with reference to the drawings. Inthe third embodiment, an image processing device having a function ofgenerating a writing log will be described as another example of the useof a detection result of the writing portion obtained as in the firstembodiment or the first to third modifications described above.

Configuration Example of Image Processing Device

FIG. 13 is a block diagram illustrating an example of a functionalconfiguration of an image processing device 10 b according to the thirdembodiment of the present disclosure.

As illustrated in FIG. 13 , the image processing device 10 b isconnected to one or more imaging devices 20, input/output devices 30,and recording devices 40, and is configured as a part of an imageprocessing system 3 of the third embodiment. The image processing device10 b of the third embodiment is also a computer including, for example,a CPU, a ROM, a RAM, and an I/O port.

In addition, the image processing device 10 b includes a writing loggeneration unit 15 b in addition to a moving image input unit 11, acontrol information input unit 12, a movement detection unit 13, awriting portion detection unit 14, and an output unit 16, which are thesame functional configuration as that of the image processing device 10of the first embodiment described above.

However, the image processing device 10 b may include, instead of themovement detection unit 13 that detects the movement of the boardsurfaces 210 a to 210 d using straight line detection, the movementdetection unit of the first modification or the second modification ofthe first embodiment described above that detects the movement of theboard surfaces 210 a to 210 d using optical flow, general objectrecognition, or the like.

The writing log generation unit 15 b acquires a movement detectionresult of the board surfaces 210 a to 210 d from the movement detectionunit 13, acquires the detection result of the writing portion by thewriting portion detection unit 14 in the frame immediately before thetiming at which the movement of at least one of the board surfaces 210 ato 210 d has been detected, and stores the detection result as thewriting log. The writing log generation unit 15 b adds the writing loggenerated in this manner to the moving image data acquired from themoving image input unit 11 and passes the writing log to the output unit16. As a result, the user can view the moving image data to which thewriting log is added with the input/output device 30.

Example of Generation Operation of Writing Log

Next, generation operation of the writing log by the image processingdevice 10 b of the third embodiment will be described with reference toFIG. 14 .

FIG. 14 is a diagram illustrating a generation result of the writing logby the image processing device 10 b according to the third embodiment ofthe present disclosure, together with moving image data. FIG. 14illustrates a frame of the moving image data at each of times t1, t2,t3, t4, and t5 and the generation result of the writing log at apredetermined timing among times t1, t2, t3, t4, and t5.

At time t1 in FIG. 14 , the writer 300 is writing an explanation contentor the like exclusively using the board surface 210 a among the boardsurfaces 210 a to 210 d.

At time t2 in FIG. 14 , the writer 300 has completed writing on theboard surface 210 a and is performing replacement by raising andlowering the board surfaces 210 a and 210 b. The movement detection unit13 detects the movement of the board surfaces 210 a and 210 b.

When the movement of the board surfaces 210 a and 210 b is detected, thewriting log generation unit 15 b acquires the detection result of thewriting portion 250 in the frame immediately before the movementdetection from the writing portion detection unit 14. In the frameimmediately before the movement of the board surfaces 210 a and 210 b isdetected, since the movement does not occur in any of the board surfaces210 a to 210 d, the writing portion detection unit 14 detects and holdsthe writing portion 250 written on the board surface 210 a.

The writing log generation unit 15 b generates the writing log 170including the writing portion 250 written on the board surface 210 afrom the detection result of the writing portion 250 acquired from thewriting portion detection unit 14, and associates the writing log withthe frame at time t2.

At time t3 in FIG. 14 , the writer 300 is writing an explanation contentor the like exclusively using the board surface 210 b.

At time t4 in FIG. 14 , the writer 300 has completed writing on theboard surface 210 b and is writing an explanation content or the likeexclusively using the board surface 210 c.

At time t5 in FIG. 14 , the writer 300 has completed writing on theboard surface 210 c and is performing replacement by raising andlowering the board surfaces 210 c and 210 d. The movement detection unit13 detects the movement of the board surfaces 210 c and 210 d.

When the movement of the board surfaces 210 c and 210 d is detected, thewriting log generation unit 15 b acquires the detection result of thewriting portion 250 in the frame immediately before the movementdetection from the writing portion detection unit 14. In the frameimmediately before the movement of the board surfaces 210 c and 210 d isdetected, since the movement does not occur in any of the board surfaces210 a to 210 d, the writing portion detection unit 14 detects and holdsthe writing portion 250 written on each of the board surfaces 210 a, 210b, and 210 c.

The writing log generation unit 15 b generates the writing log 170including the writing portion 250 written on each of the board surfaces210 a, 210 b, and 210 c from the detection result of the writing portion250 acquired from the writing portion detection unit 14, and associatesthe writing log with the frame at time t5.

As described above, the writing log generation unit 15 b generates thewriting log 170 at times t2 and t5 by using the movement detection ofthe individual board surfaces 210 a to 210 d by the movement detectionunit 13 as a trigger.

Here, it is considered that the timing at which the writer 300 or thelike moves a predetermined board surface 210 is often the timing atwhich writing on the board surface 210 has been completed. Therefore,the writing log generated at the timing of the movement of theindividual board surfaces 210 a to 210 d can be used as, for example, arecord such as lecture notes, meeting minutes, or the like in which thewriting content on the board surfaces 210 a to 210 d is recorded.

Processing Example of Image Processing Device

Next, an example of image processing by the image processing device 10 bof the third embodiment will be described with reference to FIG. 15 .FIG. 15 is a flowchart illustrating an example of a procedure for imageprocessing by the image processing device 10 b according to the thirdembodiment of the present disclosure.

In the flowchart in FIG. 15 , processing in Steps S101 to S114 is thesame as processing in Steps S101 to S114 in FIG. 8 of the firstembodiment described above, except for newly inserted Step S102 b.

That is, in a case where the movement detection of the board surfaces210 a to 210 d by the movement detection unit 13 (Step S101) does notdetect the movement of all or some of the board surfaces 210 (Step S102or Step S103: No), and there is no reference frame (Step S106: No)before the registration of the first reference frame in the initialstage of processing by the image processing device 10 a or immediatelyafter the reference frame is deleted because the movement of the boardsurfaces 210 a to 210 d occurs in the previous frame, the writingportion detection unit 14 registers the frame of the moving image dataat that time as the reference frame (Step S114). Then, processingreturns to the beginning again, for example, in the next frame.

In a case where there is the reference frame (Step S106: Yes) in theabove case (Step S102 or Step S103: No), the writing portion detectionunit 14 detects the writing portion, and reflects the holding anddeleting results for the individual set portions determined to be thewriting portion in the frame to be processed to update the referenceframe (Steps S107 to S113). Then, processing returns to the beginningagain, for example, in the next frame.

In a case where the movement of any of the board surfaces 210 a to 210 dhas been detected (Step S102: Yes), and the region to be processed fromnow on is the region corresponding to the board surface 210 where thecurrent movement has occurred (Step S103: Yes), the writing portiondetection unit 14 deletes the writing portion held in the regioncorresponding to the board surface 210 where the current movement hasoccurred in the previous frame and the reference frame of the boardsurface 210 where the current movement has occurred (Steps S104 toS105).

On the other hand, when the movement of any of the board surfaces 210 ato 210 d is detected (Step S102: Yes), the writing log generation unit15 b generates the writing log (Step S102 b).

Specifically, the writing log generation unit 15 b acquires thedetection result of the writing portion in the frame immediately beforethe movement of any of the board surfaces 210 a to 210 d is detected. Inaddition, the writing log generation unit 15 b extracts the writingportion from the acquired detection result of the writing portion andgenerates the writing log. Furthermore, the writing log generation unit15 b stores the generated writing log in association with the frame atthat time.

Then, image processing by the image processing device 10 b of the thirdembodiment ends.

Summary

According to the image processing device 10 b of the third embodiment,the writing log is generated when the movement detection unit 13 detectsthe movement of a predetermined board surface 210. As a result, therecord of the writing content on the board surfaces 210 a to 210 d canbe kept.

In the third embodiment described above, the image processing device 10b has the function of generating the writing log. However, the imageprocessing device having the writing log generation function may havethe function of generating the composite moving image data bysuperimposing the writing portion on the moving image data, as in theimage processing device 10 a of the second embodiment described above.In this case, the generated writing log is stored in association withthe corresponding frame of the composite moving image data, for example.

First Modification

Next, an image processing device of a first modification of the thirdembodiment will be described. The image processing device of the firstmodification is different from the third embodiment described above inthat the image processing device determines whether or not to generatethe writing log according to a movement speed of the board surfaces 210a to 210 d.

A writing log generation unit of the image processing device of thefirst modification acquires information on the movement speed of theboard surface 210 whose movement has been detected, together with amovement detection result of the board surfaces 210 a to 210 d, from amovement detection unit of the first modification. In addition, thewriting log generation unit of the first modification determines whetheror not to generate the writing log based on the information of themovement speed of the board surface 210.

For example, in a case where the movement speed of the board surface 210is less than a predetermined value, the writing log generation unit ofthe first modification can be prevented from generating the writing log.Then, in a case where the movement speed of the board surface 210 isequal to or greater than the predetermined value, the writing loggeneration unit of the first modification can generate the writing log.

The user performs the setting as described above in advance on the imageprocessing device of the first modification. When moving a predeterminedboard surface 210 during recording of the moving image data, the writer300 or the like may slowly move the board surface 210 at the timing atwhich the writing log is considered to be unnecessary. On the otherhand, the writer 300 may quickly move the board surface 210 at thetiming at which the writing log is considered to be necessary.

According to the image processing device of the first modification, thewriting log is generated when the movement detection unit of the firstmodification detects the movement of the board surface at apredetermined speed or more. As a result, for example, the writing logcan be generated at a timing desired by the writer 300, and a moreappropriate record can be kept.

Second Modification

Next, an image processing device of a second modification of the thirdembodiment will be described. The image processing device of the secondmodification is different from the third embodiment described above inthat the image processing device determines whether or not to generatethe writing log according to a progress status of writing on the boardsurface 210.

When the movement of a predetermined board surface 210 is detected, awriting log generation unit of the image processing device of the secondmodification acquires a detection result of a writing portion in a frameimmediately before movement detection and a plurality of frames beforethe frame immediately before the movement detection, from a writingportion detection unit of the second modification.

In addition, the writing log generation unit of the second modificationdetermines a writing direction of the writing portion on the boardsurface 210 whose movement has been detected from the detection resultof the writing portion in the plurality of frames including the frameimmediately before the movement detection of the board surface 210. Asthe writing direction, a case where the writing portion proceeds fromtop to bottom, for example, in the case of horizontal writing of thewriting portion or the like, and a case where the writing portionproceeds from right to left, for example, in the case of verticalwriting of the writing portion are considered.

In addition, the writing log generation unit of the second modificationdetermines whether or not the writing portion detected in the frameimmediately before the movement detection of the board surface 210reaches from one end to the other end in the writing direction of theboard surface 210.

That is, for example, in the case where the writing direction is fromtop to bottom, it is determined whether or not the writing portionreaches from the upper end to the lower end of the board surface 210. Onthe other hand, for example, in the case where the writing direction isfrom right to left, it is determined whether or not the writing portionreaches from the right end to the left end of the board surface 210.

Whether or not the writing portion reaches from one end to the other endof the board surface 210 can be determined based on, for example,information on a vertical width or a horizontal width of the boardsurface 210 or the like given from the user.

The writing log generation unit of the second modification generates thewriting log in a case where the writing portion reaches from one end tothe other end of the board surface 210. The writing log generation unitof the second modification does not generate the writing log in a casewhere the writing portion does not reach from one end to the other endof the board surface 210.

Here, in the case where the writing portion reaches from one end to theother end of the board surface 210, it can be considered that writing onthe board surface has been completed. On the other hand, in a case wherethe writing portion does not reach from one end to the other end of theboard surface 210, the writer 300 can still add writing on the boardsurface 210, and there is a possibility that additional writing will beperformed to the writing portion, for example, by pulling back the boardsurface 210 that has been temporarily moved upward.

Therefore, by adding the condition that the writing portion reaches fromone end to the other end of the board surface 210 to the writing loggeneration conditions, it is possible to generate the writing log at amore accurate timing at which the writing on the board surface 210 hasbeen completed.

According to the image processing device of the second modification, thewriting log is generated when the movement detection unit of the secondmodification detects the movement of the board surface 210 and thewriting portion reaches the other end from one end of the board surface210. As a result, it is possible to more reliably determine the timingat which writing on the predetermined board surface 210 has beencompleted and generate the writing log at a more appropriate timing. Inaddition, even in a case where additional writing is performed on theboard surface 210 that has been pulled back in the middle of themovement, it is possible to suppress duplicated generation of thewriting log for the similar writing content.

The effects described herein are merely examples and are not subject tolimitations, and other effects may be provided.

Other Embodiments

The image processing devices of the first to third embodiments and thefirst to third modifications thereof described above are a computerincluding, for example, a CPU, a ROM, a RAM, and an I/O port. However,the image processing devices may have a configuration including anapplication specific integrated circuit (ASIC) dedicated for theapplication described above in the first to third embodiments and thefirst to third modifications thereof.

The image processing devices of the first to third embodiments and thefirst to third modifications thereof described above include anindependent device configuration separated from the imaging device 20.However, the function of the image processing device may be mounted onthe imaging device.

The image processing devices of the first to third embodiments and thefirst to third modifications thereof described above include a ROM inwhich the program related to image processing is stored. Such a programcan be provided by being stored in a recording medium or the like so asto be readable by a computer, for example. The recording medium is, forexample, a magnetic disk, an optical disk, a magneto-optical disk, aflash memory, or the like. Furthermore, the program described above maybe distributed via, for example, a network without using the recordingmedium. In this case, the program may not necessarily be installed inthe image processing device.

The present technique may also have the following configurations:

-   (1) An image processing device comprising:    -   a movement detection unit that detects movement of a movable        writing medium based on moving image data obtained by imaging        the writing medium and a writer writing on the writing medium;        and    -   a writing portion detection unit that detects a writing portion        on the writing medium,    -   wherein the writing portion detection unit    -   executes processing corresponding to the movement of the writing        medium in a case where the movement of the writing medium is        detected.-   (2) The image processing device according to (1),    -   wherein the movement detection unit    -   detects a straight line portion of a frame of the writing        medium, and detects the movement of the writing medium by        detecting the movement of the straight line portion.-   (3) The image processing device according to (2),    -   wherein the movement detection unit    -   distinguishes between a straight line included in the writing        portion and the frame of the writing medium based on a length of        the straight line portion.-   (4) The image processing device according to (1),    -   wherein the movement detection unit    -   detects the movement of the writing medium by estimating an        optical flow in which movement of the writing portion associated        with the movement of the writing medium is represented by a        vector.-   (5) The image processing device according to (4),    -   wherein the movement detection unit    -   groups the optical flow and distinguishes between movement of        the writer and the movement of the writing medium based on a        size of a grouped region and a direction of movement.-   (6) The image processing device according to (1),    -   wherein the movement detection unit    -   detects the movement of the writing medium by recognizing the        writing medium using general object recognition that recognizes        an object by a general name, and tracking a position or a center        of gravity of the writing medium.-   (7) The image processing device according to (6),    -   wherein the movement detection unit    -   distinguishes between movement of the writer and the movement of        the writing medium by recognizing the writer as well using the        general object recognition.-   (8) The image processing device according to any one of (1) to (7),    -   wherein the writing portion detection unit    -   deletes a detection result of the writing portion on the writing        medium in a case where the movement of the writing medium is        detected.-   (9) The image processing device according to (8),    -   wherein the writing medium is one of a plurality of individually        movable writing media included in a writing medium group, and    -   the movement detection unit    -   individually detects movement of the plurality of writing media,        and,    -   in a case where the movement of at least one of the plurality of        writing media is detected, the writing portion detection unit    -   deletes the detection result of the writing portion on the        writing medium whose movement is detected and holds the        detection result of the writing portion on the writing medium        whose movement is not detected.-   (10) The image processing device according to any one of (1) to (9),    -   wherein the movement detection unit    -   changes an angle of view of an imaging device that images the        moving image data in a case where the writing medium moves out        of a frame of the moving image data.-   (11) The image processing device according to (10), wherein the    movement detection unit    -   causes imaging to be performed at a wider angle of view in a        case where the angle of view of the imaging device is changed.-   (12) The image processing device according to any one of (1) to (9),    further comprising    -   a data generation unit that generates writing data based on a        detection result of the writing portion.-   (13) The image processing device according to (12),    -   wherein the data generation unit    -   generates composite moving image data by superimposing the        writing data on the moving image data.-   (14) The image processing device according to any one of (1) to    (13), comprising    -   a writing log generation unit that generates writing log based        on a detection result by the movement detection unit.-   (15) The image processing device according to (14),    -   wherein the writing log generation unit    -   generates the writing log in a case where the movement detection        unit detects the movement of the writing medium.-   (16) The image processing device according to (14),    -   wherein the writing log generation unit    -   generates the writing log in a case where the movement detection        unit detects the movement of the writing medium at a        predetermined speed or more.-   (17) The image processing device according to (13),    -   wherein the writing log generation unit    -   generates the writing log in a case where the movement detection        unit detects the movement of the writing medium and the writing        portion reaches from one end to the other end of the writing        medium.-   (18) The image processing device according to any one of (1) to    (17),    -   wherein the movement detection unit    -   detects the movement of the writing medium based on information        related to the writing medium input by a user.-   (19) An image processing method comprising:    -   detecting movement of a movable writing medium based on moving        image data obtained by imaging the writing medium and a writer        writing on the writing medium; and    -   detecting a writing portion on the writing medium,    -   wherein in detecting the writing portion, processing        corresponding to the movement of the writing medium is executed        in a case where the movement of the writing medium is detected.-   (20) A program causing a computer to execute:    -   a process of detecting movement of a movable writing medium        based on moving image data obtained by imaging the writing        medium and a writer writing on the writing medium; and    -   a process of detecting a writing portion on the writing medium,    -   wherein, in the process of detecting the writing portion,    -   processing corresponding to the movement of the writing medium        is executed in a case where the movement of the writing medium        is detected.

Although several embodiments of the present disclosure have beendescribed, these embodiments have been presented as examples, and arenot intended to limit the scope of the invention. These embodiments canbe implemented in various other forms, and various omissions,substitutions, and changes can be made without departing from the gistof the invention. These embodiments and modifications thereof areincluded in the scope and gist of the invention and are included in thescope of the invention described in the claims and the equivalentthereof.

Reference Signs List 1, 2 INFORMATION PROCESSING SYSTEM 10, 10 a, 10 bINFORMATION PROCESSING DEVICE 13 MOVEMENT DETECTION UNIT 14 WRITINGPORTION DETECTION UNIT 15 a DATA GENERATION UNIT 15 b WRITING LOGGENERATION UNIT 20 IMAGING DEVICE 30 INOUT/OUTPUTDEVICE 40 RECORDINGDEVICE

1. An image processing device comprising: a movement detection unit thatdetects movement of a movable writing medium based on moving image dataobtained by imaging the writing medium and a writer writing on thewriting medium; and a writing portion detection unit that detects awriting portion on the writing medium, wherein the writing portiondetection unit executes processing corresponding to the movement of thewriting medium in a case where the movement of the writing medium isdetected.
 2. The image processing device according to claim 1, whereinthe movement detection unit detects a straight line portion of a frameof the writing medium, and detects the movement of the writing medium bydetecting the movement of the straight line portion.
 3. The imageprocessing device according to claim 2, wherein the movement detectionunit distinguishes between a straight line included in the writingportion and the frame of the writing medium based on a length of thestraight line portion.
 4. The image processing device according to claim1, wherein the movement detection unit detects the movement of thewriting medium by estimating an optical flow in which movement of thewriting portion associated with the movement of the writing medium isrepresented by a vector.
 5. The image processing device according toclaim 4, wherein the movement detection unit groups the optical flow anddistinguishes between movement of the writer and the movement of thewriting medium based on a size of a grouped region and a direction ofmovement.
 6. The image processing device according to claim 1, whereinthe movement detection unit detects the movement of the writing mediumby recognizing the writing medium using general object recognition thatrecognizes an object by a general name, and tracking a position or acenter of gravity of the writing medium.
 7. The image processing deviceaccording to claim 6, wherein the movement detection unit distinguishesbetween movement of the writer and the movement of the writing medium byrecognizing the writer as well using the general object recognition. 8.The image processing device according to claim 1, wherein the writingportion detection unit deletes a detection result of the writing portionon the writing medium in a case where the movement of the writing mediumis detected.
 9. The image processing device according to claim 8,wherein the writing medium is one of a plurality of individually movablewriting media included in a writing medium group, and the movementdetection unit individually detects movement of the plurality of writingmedia, and, in a case where the movement of at least one of theplurality of writing media is detected, the writing portion detectionunit deletes the detection result of the writing portion on the writingmedium whose movement is detected and holds the detection result of thewriting portion on the writing medium whose movement is not detected.10. The image processing device according to claim 1, wherein themovement detection unit changes an angle of view of an imaging devicethat images the moving image data in a case where the writing mediummoves out of a frame of the moving image data.
 11. The image processingdevice according to claim 10, wherein the movement detection unit causesimaging to be performed at a wider angle of view in a case where theangle of view of the imaging device is changed.
 12. The image processingdevice according to claim 1, further comprising a data generation unitthat generates writing data based on a detection result of the writingportion.
 13. The image processing device according to claim 12, whereinthe data generation unit generates composite moving image data bysuperimposing the writing data on the moving image data.
 14. The imageprocessing device according to claim 1, comprising a writing loggeneration unit that generates writing log based on a detection resultby the movement detection unit.
 15. The image processing deviceaccording to claim 14, wherein the writing log generation unit generatesthe writing log in a case where the movement detection unit detects themovement of the writing medium.
 16. The image processing deviceaccording to claim 14, wherein the writing log generation unit generatesthe writing log in a case where the movement detection unit detects themovement of the writing medium at a predetermined speed or more.
 17. Theimage processing device according to claim 14, wherein the writing loggeneration unit generates the writing log in a case where the movementdetection unit detects the movement of the writing medium and thewriting portion reaches from one end to the other end of the writingmedium.
 18. The image processing device according to claim 1, whereinthe movement detection unit detects the movement of the writing mediumbased on information related to the writing medium input by a user. 19.An image processing method comprising: detecting movement of a movablewriting medium based on moving image data obtained by imaging thewriting medium and a writer writing on the writing medium; and detectinga writing portion on the writing medium, wherein in detecting thewriting portion, processing corresponding to the movement of the writingmedium is executed in a case where the movement of the writing medium isdetected.
 20. A program causing a computer to execute: a process ofdetecting movement of a movable writing medium based on moving imagedata obtained by imaging the writing medium and a writer writing on thewriting medium; and a process of detecting a writing portion on thewriting medium, wherein, in the process of detecting the writingportion, processing corresponding to the movement of the writing mediumis executed in a case where the movement of the writing medium isdetected.